Retrieval-based Question Answering for Machine Reading Evaluation

The Question Answering for Machine Reading (QA4MRE) task was set up as a reading comprehension test consisting of 120 multiple-choice questions pertaining to twelve target texts (the test documents) grouped in three different topics. Since this is the first year that we participate in the task, we decided to follow a relatively knowledge-poor approach that is mainly based on Information Retrieval (IR) techniques. We participated in the English task only. In its most basic version, our system takes the question as a query and uses a standard word-based ranking model to retrieve the most relevant fragments from the test document. Then it matches each of the multiple-choice answer candidates against the set of retrieved text fragments to select the answer with the highest summed similarity (again measured using a standard ranking model). We investigated two forms of information expansion to improve over this baseline: (1) statistical expansion of the test document with sentences from the topical background corpus, and (2) expansion of the question with automatically gathered facts from the background corpus. Our best-performing experimental setting reaches an overall c@1 score of 0.37. We found that statistical expansion of the test documents gives very different results for the three topics but overall it gives an improvement over the document-only baseline. We could not gain any improvements from question-to-facts expansion. More experiments are needed to find a good implementation of fact expansion. In the near future, we will follow up on the current work with more experiments related to expansion of questions and documents for the purpose of question answering.